Modeling the Interaction Between Speech and Gesture

Until now theories of the gesture-speech relationship have been difficult to evaluate because of their descriptive basis. In this paper we provide a tool for investigating the relationship between speech and gesture: a system that generates speech, intonation, and gesture using two copies of an identical program that have different knowledge of the world and must cooperate to accomplish a goal. The output of the dialogue generation is fed into a three-dimensional interactive animated model – two graphic figures on a computer screen who gesture according to the rules given to the system. The advantage of computer modeling in this domain is that it forces us to come up with predictive theories of the gesture-speech relationship. A felicitous outcome is a working system to realize autonomous animated conversational agents, for virtual reality and other purposes. Areas: discourse, gesture, animation Presentation: talk This research is partially supported by NSF VPW GER-9350179; NSF IRI91-17110; NSF graduate fellowships; ARO Grant DAAL03-89-C-0031 including participation by the U.S. Army Research Laboratory (Aberdeen); U.S. Air Force DEPTH contract through Hughes Missile Systems F33615-91-C-000; DMSO through the University of Iowa; National Defense Science and Engineering Graduate Fellowships; Naval Training Systems Center N61339-93M-0843; Sandia Labs AG-6076; NASA KSC NAG10-0122; MOCO, Inc.; and NSF Instrumentation and Laboratory Improvement Program Grant USE-9152503.